1.1 Textbook Learning Objectives

• Differentiate between Type I and Type II Errors
• Describe hypothesis testing in general and in practice
• Conduct and interpret hypothesis tests for a single population mean, population standard deviation known.
• Conduct and interpret hypothesis tests for a single population mean, population standard deviation unknown.
• Conduct and interpret hypothesis tests for a single population proportion.

1.2 Instructor Learning Objectives

• Be able to identify different parts of a hypothesis given a narrative example
• Understand the connection between \(\alpha\), p-values, probability, and Type I and II errors
• Be able to work through practical inferential statistics computations for a single sample scenario

1.3 Introduction

• In statistics, hypothesis testing is the process by which we evaluate whether data supports making a certain conclusion beyond a reasonable doubt

• We have certain steps to work through a hypothesis test:

  • Set up the contradictory null and alternative hypotheses
  • Collect data in a sample
  • Determine an appropriate inferential test and distribution to represent our variables
  • Conclude whether we can reject the null hypothesis or if we cannot

• Important: I think it's important to emphasize that it is not our 'goal' to reject the null hypothesis - as empiricists, our orientation should be towards truth, not a certain outcome

2.1 Introduction

• A hypothesis is just a prediction or a statement of a possible outcome
  • E.g., I hypothesize (or predict) that college students who are Gen Z have significantly worse attention than Millennial college students
  • Most scientific studies begin with some hypothesis of what outcomes will look like

• Discuss: Consider that you are a middle-school teacher being asked by administration to implement an especially permissive attendance structure, write a hypothesis of what this will do to student math scores

• When we make hypotheses in statistics, we do so in a contradictory pair by making a null hypothesis and an alternative hypothesis
  • In frequentist statistical testing, we frame our inferential tests as helping us determine whether to reject the null - there is a whole other field of statistics called Bayesian statistics that has a somewhat different focus

• Important: It's important to understand that, as part of the statistical testing process, we should always consider what our null and alternative hypothesis is!

2.2 Null Hypotheses

• A null hypothesis is a prediction or statement of no difference between two or more things
  • It is usually written as \(H_0\)
  • Opposite of alternative hypothesis
• A null hypothesis is often written to include an equal sign such as with:
  • \(=\)
  • But also, \(\geq\) & \(\leq\)
• Example: There is no difference in attention between Gen Z and Millennial college students
  • Represented in notation \(H_0: Attention_{Z} = Attention_{Millennial}\)

• Discuss: Write the null hypothesis for comparing mental health between women and men in the legal field, where they have equal levels of depression

2.3 Alternative Hypotheses

• Our alternative hypothesis suggests a difference between two or more things
  • It is most often represented as \(H_A\) or \(H_1\)
  • Opposite of Null Hypothesis
  • This is usually close to how we would phrase a study hypothesis
• The alternative hypothesis does not contain equal signs
  • This would include \(\neq\), \(>\), and \(<\)
• Example: There is a difference between Gen Z and Millennial college students in attention
  • Represented in notation \(H_A: Attention_{Z} \neq Attention_{Millennial}\)

• Discuss: Write the alternative hypothesis for comparing mental health between women and men in the legal field, where women have higher levels of depression

3.1 Decision-making

• When we gather and analyze the data we gather from a sample, we use those results to make a decision
  • The decision comes down to whether we can reject the null hypothesis or not (sometimes called ‘retaining’ the null hypothesis)
  • Ideally we avoid making a type I or type II error with our decision

3.2 Types of Errors

• A type I error is when we decide to reject the null hypothesis, when it is actually true
  • The probability of a Type I error occurring (i.e., \(P(TypeI)\)) is given as \(\alpha\)

• Discuss: Review: where did we come across alpha before? Explain it's significance in the context of module 8!

• A type II error is when we decide to retain the null hypothesis when it is actually false
  • The probability of a Type II error occurring (i.e., \(P(TypeII)\)) is given as \(\beta\)

• Important: When I was first trained on statistics, I was taught to view the Type I error as being more 'egregious' - 'the number one error to avoid!' - but realistically, they both result in flawed conclusions which could be dangerous

• Ideally, both \(\alpha\) and \(\beta\) should be as small as possible, as to try an avoid making an error of any sort!
  • An extension of this is that power, the likelihood we correctly reject the null when it is false
  • \(Power = 1 - \beta\)

• Question: In a hypothetical experiment when comparing students before and after a new instructional style, I correctly decide I can reject the null hypothesis that there was no change. This is accurate, as there actually was a change. What error occurred here?
• A) No
• B) error
• C) []{.quarto-shortcode__-param
• D) "

4.1 Introduction

• Its important that we identify the sampling probability distribution we can use to represent a certain variable and it’s mean or proportion
  • Effectively, this is where we are choosing a test - a very important step in our analysis!

• Important: Choosing the 'right' test for certain situations is probably the most valuable skill a statistician has!

• Discuss: Review: What is the formula for the SE of the mean and how does that tie back to the normal distribution of sample means?

4.2 Assumptions

• An assumption is some prerequisite of the distribution we are using

• Take for example: the z-test for a population mean with the normal distribution. We need:

  • \(\mu\), \(\sigma\), and the population must be normally distributed / large enough to appeal to the CLT
  • We must also have a simple random sample from the population

• Discuss: Review: Describe why a simple random sample is often impractical. This is where research design intersects with analysis!

• The t-test for a population mean with the t-distribution assumes we also have a simple random sample from a normally distributed population

• Important: Different inferential tests have different assumptions, and violating those assumptions can have big consequences! Too many researchers outright ignore these for convenience

5.1 Introduction

• While accounting for the information, statistics, parameters, and sampling distribution we have do help us make good choices about hypothesis testing - they don’t tell us everything
  • We always have to remember that things in statistics are probabilistic!
  • With our sampling and data gathering we may run into a rare event
• To account for the possibility of a rare event, we test the null hypothesis somehow

• Discuss: Try re-explaining, in your own words, what 'probablistic' means and why it's readily applicable to the practice of statistics

5.2 Using the Sample to Test the Null Hypothesis

• This is where we introduce the p-value, which is the chance that, under the null hypothesis being true, our results will be as extreme as they are
  • Example: a test result returns a p-value of 0.07. Assuming the null hypothesis is true, this result only had a 7% chance of occurring.
• Effectively, when we test against the null hypothesis and determine a p-value, we are trying to gauge how likely our results were to occur if the null hypothesis was true
  • If our results are especially rare under the null hypothesis (i.e., a low p-value), then we may be inclined to believe that our case is somehow truly different, and thus the null hypothesis is incorrect and can be rejected

• Important: There are many misunderstandings people have about statistics, but failing to understand the meaning of p-values is probably the single most common and pervasive errors people make in interpreting statistics.

5.3 Decision and Conclusion

• How do we determine if the p-value is, “rare enough”?
  • Ideally, we test it against a preset/preconceived significance level, also given as \(\alpha\).
  • We decide whether our p-value is \(\geq\) or \(<\) our \(\alpha\)

• Important: This is the exact same alpha that showed up in regard to confidence level in the last module and earlier in our Type I error! We *choose* what we wish our significance level to be at the onset of our analyses.

• If you don’t see further information, the most common significance level is \(\alpha = 0.05\)
  • However, this isn’t a hard set rule.

• Discuss: Take a look at the prior discussion on Type I and Type II error - try using those terms to discuss why have too high of an alpha level might be risky

5.4 Tails of a Test

• A test may be described as two-tailed, left-tailed, or right-tailed, dependent on the sign used in the alternative hypothesis
  • \(H_a: P > 0.5 \rightarrow\) right-tailed (one-tailed)
  • \(H_a: P < 0.5 \rightarrow\) left-tailed (one-tailed)
  • \(H_a: P \neq 0.5 \rightarrow\) two-tailed
• This needs to be set as part of the study set up, not during analysis!

• Discuss: Consider the following example: Johnny predicts that Samantha has more money than Becky right now. Is this a one-tailed or two-tailed test and why?

6.1 Recap

• Hypothesis testing is the backbone of inferential statistics, and testing against the null hypothesis is how we determine whether our results were just a chance “fluke” or likely some real difference!

• Proper hypothesis testing hinges upon wise identification of what values we have and what distribution best describes our variables. Each distribution and test we could use has a set of assumptions we need to be aware of

• P-values, alpha, beta, and the two hypotheses are all connected, and we must be careful to make the correct decisions - wary of the relative risk of Type I and Type II errors

6.2 Lecture Check-in

• Make sure to complete any lecture check-in tasks associated with this lecture!

Module 9 Lecture - Repeated Measures ANOVA || Analysis of Variance